System, apparatus, method, and program for displaying wide view image

ABSTRACT

An image display system includes an image acquisition unit which acquires an image, an area setting unit which sets a plurality of viewing areas to which the image is to be assigned according to a direction of the image, an image assignment unit which assigns the image to each of the plurality of viewing areas, an operation determination unit which discriminates an operation in a first viewing area among the plurality of viewing areas each having the image assigned thereto, and a display control unit which displays the image in the first viewing area and the image in a second viewing area other than the first viewing area among the plurality of viewing areas concurrently based on a result of the discrimination by the operation determination unit.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation application and claimspriority under 35 U.S.C 120 to U.S. patent application Ser. No.15/527,058 filed on May 16, 2017, which is the National Stage ofInternational Application No. PCT/JP2016/000531 filed on Feb. 2, 2016,which claims priority to Japanese Patent Application No. 2015-023176,filed on Feb. 9, 2015. The contents of the applications are incorporatedherein by reference in their entirety.

TECHNICAL FIELD

The present invention relates to an image display system, an informationprocessing apparatus, an image display method, an image display program,an image processing apparatus, an image processing method, and an imageprocessing program.

BACKGROUND ART

Imaging and displaying of wide view images, including panoramic images(images with a wide field of view in a horizontal plane) andomnidirectional images (images with a field of view covering alldirections), are becoming widespread. For example, see JapaneseLaid-Open Patent Publication No. 2011-076249.

In a case of displaying a wide view image having a field of viewcovering all directions on a screen, if the wide view image is displayedin a viewing area of the screen, the visibility is lowered. A method forsolving the problem is that partial images with field of views coveringmutually different directions are displayed in a plurality of viewingareas.

When displaying the partial images of the different directions in theplurality of viewing areas, an image of a predetermined direction may befixed in several of the viewing areas. In this case, operations ofmoving and resizing the image are not performed with respect to suchviewing areas.

A method of enabling the moving and resizing operations to be performedwith respect to a viewing area is conceivable. However, it is preferredthat the moving and resizing operations can be performed with respect tothe viewing areas independently of each other.

When a wide view image is displayed in a plurality of viewing areas,there is no user interface which provides appropriate operations for thecharacteristics of the wide view image. The problem is that a user isunable to perform a desired operation on the wide view image.

RELATED-ART DOCUMENT Patent Document

[Patent Document 1] Japanese Laid-Open Patent Publication No.2011-076249

SUMMARY OF INVENTION

In one aspect, the present invention provides a user interface whichenables flexible operations to be performed when displaying an image ina plurality of viewing areas.

In one embodiment, the present invention provides an image displaysystem including one or more processors, the processor including animage acquisition unit configured to acquire an image; an area settingunit configured to set a plurality of viewing areas to which the imageis to be assigned according to a direction of the image; an imageassignment unit configured to assign the image to each of the pluralityof viewing areas; an operation determination unit configured todiscriminate an operation in a first viewing area among the viewingareas each having the image assigned thereto; and a display control unitconfigured to display the image in the first viewing area and the imagein a second viewing area other than the first viewing area among theviewing areas concurrently based on a result of the discrimination bythe operation determination unit.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims. It is to be understood that both the foregoing generaldescription and the following detailed description are exemplary andexplanatory and are not restrictive of the invention as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an example of an image display systemaccording to an embodiment;

FIG. 2A is a diagram showing an outer appearance of an imaging device;

FIG. 2B is a diagram showing an outer appearance of the imaging device;

FIG. 2C is a diagram showing an outer appearance of the imaging device;

FIG. 3 is a block diagram showing a hardware configuration of theimaging device;

FIG. 4 is a block diagram showing a hardware configuration of aninformation processing apparatus;

FIG. 5 is a diagram showing a software configuration of the imagedisplay system;

FIG. 6 is a diagram showing an example of a viewing area pattern table;

FIG. 7 is a diagram showing an example of an arrangement pattern ofviewing areas;

FIG. 8 is a diagram showing an example of the arrangement pattern ofviewing areas;

FIG. 9 is a diagram showing an example of the arrangement pattern ofviewing areas;

FIG. 10 is a diagram showing an example of the arrangement pattern ofviewing areas;

FIG. 11 is a diagram showing an example of the arrangement pattern ofviewing areas;

FIG. 12 is a diagram showing an example of the arrangement pattern ofviewing areas;

FIG. 13 is a diagram showing an example of the arrangement pattern ofviewing areas;

FIG. 14 is a diagram showing an example of an area transition patterntable;

FIG. 15 is a diagram showing an example of current data;

FIG. 16 is a diagram showing an imaging action performed with theimaging device;

FIG. 17 is a diagram showing an omnidirectional image produced by theimaging device;

FIG. 18 is a diagram showing an example of generation of anomnidirectional image;

FIG. 19A is a diagram showing a relationship between an X-axis, aY-axis, and a Z-axis;

FIG. 19B is a diagram showing an arrangement of a virtual camera forcapturing a cutout of a display image;

FIG. 19C is a diagram showing a field of view of the virtual camera;

FIG. 20 is a diagram for explaining a cutout of a display image;

FIG. 21 is a flowchart for explaining a process performed by theinformation processing apparatus;

FIG. 22A is a diagram showing an example of operations and displaychanges;

FIG. 22B is a diagram showing an example of operations and displaychanges;

FIG. 23A is a diagram showing an example of operations and displaychanges;

FIG. 23B is a diagram showing an example of operations and displaychanges;

FIG. 24A is a diagram showing an example of operations and displaychanges;

FIG. 24B is a diagram showing an example of operations and displaychanges;

FIG. 25A is a diagram showing an example of operations and displaychanges;

FIG. 25B is a diagram showing an example of operations and displaychanges;

FIG. 26A is a diagram showing an example of operations and displaychanges;

FIG. 26B is a diagram showing an example of operations and displaychanges;

FIG. 27A is a diagram showing an example of operations and displaychanges;

FIG. 27B is a diagram showing an example of operations and displaychanges;

FIG. 28A is a diagram showing an example of operations and displaychanges;

FIG. 28B is a diagram showing an example of operations and displaychanges;

FIG. 29A is a diagram showing an example of operations and displaychanges;

FIG. 29B is a diagram showing an example of operations and displaychanges;

FIG. 30A is a diagram showing an example of operations and displaychanges;

FIG. 30B is a diagram showing an example of operations and displaychanges;

FIG. 31A is a diagram showing an example of operations and displaychanges; and

FIG. 31B is a diagram showing an example of operations and displaychanges.

DESCRIPTION OF EMBODIMENTS

A description will be given of embodiments with reference to theaccompanying drawings.

FIG. 1 shows an image display system 10 according to an embodiment. Asshown in FIG. 1, the image display system 10 includes an imaging device1, an information processing apparatus 2, and a web server 3. In theimage display system 10, the imaging device 1 and the informationprocessing apparatus 2 are interconnected by wired or wirelesscommunication, and the web server 3 and the information processingapparatus 2 are interconnected by wired or wireless communication.

The imaging device 1 is configured to take an omnidirectional image andsupply the omnidirectional image to the information processing apparatus2. Note that the image taken by the imaging device 1 is not limited tothe omnidirectional image and may be a panoramic image or any otherimage. The information processing apparatus 2 may be an apparatus, suchas a smart phone, a tablet, a mobile phone, or a personal computer (PC).The information processing apparatus 2 is configured to acquire an imagefrom the imaging device 1 or the web server 3 and display the image onthe information processing apparatus 2, which is viewed by a user. Theweb server 3 is configured to acquire and accumulate images beforehandand supply the images to the information processing apparatus 2.

FIGS. 2A to 2C are diagrams showing an outer appearance of the imagingdevice 1. FIG. 2A shows a state in which a user holds the imaging device1 by hand. FIG. 2B is a right-hand side view of the imaging device 1shown in FIG. 2A. FIG. 2C is a top view of the imaging device 1 shown inFIG. 2A.

The imaging device 1 includes a front imaging element 1H1 disposed at anupper end portion of one surface of a casing, and a rear imaging element1H2 disposed at an upper end portion of the opposite surface of thecasing. Further, the imaging device 1 includes a switch 1H3 disposed ata central portion of the casing. The switch 1H3 is provided to controlthe imaging device 1 to perform imaging upon depression of the switch1H3.

FIG. 3 shows a hardware configuration of the imaging device 1. As shownin FIG. 3, the imaging device 1 includes an imaging unit 1H4, an imageprocessing unit 1H7, and an imaging control unit 1H8. The imaging unit1H4 includes a front imaging element 1H1, a rear imaging element 1H2, alens 1H5 disposed at the front imaging element 1H1, and a lens 1H6disposed at the rear imaging element 1H2. The image processing unit 1H7is configured to generate an omnidirectional image based on the imagestaken by the front imaging element 1H1 and the rear imaging element 1H2.The imaging control unit 1H8 is configured to control imaging of each ofthe front imaging element 1H1 and the rear imaging element 1H2.

Further, the imaging device 1 includes a central processing unit (CPU)1H9, a read-only memory (ROM) 1H10, a static random access memory (SRAM)1H11, a dynamic random access memory (DRAM) 1H12, an operation interface1H13, a network interface 1H14, a radio interface 1H15, and an antenna1H16, which are interconnected by a bus 1H17. These elements areprovided to perform a process for supplying a taken omnidirectionalimage to an external device, and to control the image processing unit1H7 and the imaging control unit 1H8.

FIG. 4 shows a hardware configuration of the information processingapparatus 2. As shown in FIG. 4, the information processing apparatus 2includes an auxiliary storage device 2H1, a main storage device 2H2, aninput/output device 2H3, a state sensor 2H4, a CPU 2H5, and a networkinterface 2H6, which are interconnected by a bus 2H7.

The state sensor 2H4 includes an angle sensor and an accelerationsensor, and these sensors are provided to detect a portrait orientationor a landscape orientation of the information processing apparatus 2.The input/output device 2H3 includes a display unit for displaying ascreen, and a touch panel for receiving a user operation. The CPU 2H5 isa processor configured to control acquisition of an image from theimaging device 1 or the web server 3, and displaying of an image in ascreen.

FIG. 5 shows a software configuration of the image display system 10. Asshown in FIG. 5, the imaging device 1 includes a first imaging unit 11,a second imaging unit 12, and an omnidirectional image generation unit13. The first imaging unit 11 corresponds to the hardware of the frontimaging element 1H1 (FIG. 3), and the second imaging unit 12 correspondsto the hardware of the rear imaging element 1H2 (FIG. 3). Theomnidirectional image generation unit 13 corresponds to the hardware ofthe image processing unit 1H7 (FIG. 3).

As shown in FIG. 5, the information processing apparatus 2 includes animage acquisition unit 21, an area control unit 22, an input/output unit23, and a storage unit 24. The area control unit 22 includes an imageselection unit 221, an area setting unit 222, an image assignment unit223, an operation determination unit 224, and an image operation unit225. The storage unit 24 includes an image data storage unit 241, aviewing area pattern storage unit 242, an area transition patternstorage unit 243, and a current data storage unit 244.

The image acquisition unit 21 is configured to acquire an image from theimaging device 1 or the web server 3, and store the image into the imagedata storage unit 241.

The area control unit 22 is configured to perform a process for managingviewing areas of a screen, for displaying an image in the viewing areasof the screen, for receiving from a user an operation (moving, resizing,etc.) on the displayed image, and for reflecting the user operation inthe displayed image.

The image selection unit 221 is configured to prompt a user to select animage to be displayed from among the images stored in the image datastorage unit 241. The image selection unit 221 is configured to store animage ID which identifies the selected image into the current datastorage unit 244 as a current data item.

The area setting unit 222 is configured to receive from a user aselection of a viewing area pattern from among the viewing area patternsstored in the viewing area pattern storage unit 242, and set the viewingareas according to the selected viewing area pattern. The area settingunit 222 is configured to store data of the selected viewing areapattern and data of a corresponding area transition pattern (which isstored in the area transition pattern storage unit 243 when the useroperation covers two or more viewing areas) into the current datastorage unit 244 as current data items.

The image assignment unit 223 is configured to acquire an image from theimage data storage unit 241 and assign the image to the viewing areasset by the area setting unit 222. The image assigned to the viewingareas may include image portions in different viewing areas of theacquired image and may cover a partially overlapping viewing area.

The operation determination unit 224 is configured to determine a useroperation on the image. At this time, the operation determination unit224 refers to the operation coordinate information received from theinput/output unit 23 and to the current data stored in the current datastorage unit 244, in order to determine which viewing area of theviewing areas corresponds to the image on which the user operation isperformed, and determine whether the user operation is an operationcovering two or more viewing areas. Note that the user operation on theimage may include an operation to touch the screen by a finger and anoperation to touch the screen by an electronic pen, a mouse, etc.

The image operation unit 225 is configured to reflect the user operationin the image data of the image data storage unit 241 based on theoperation determined by the operation determination unit 224. Further,the image operation unit 225 is configured to reflect, in the currentdata of the current data storage unit 244, a direction and a viewingangle of the image having been varied by the user operation.

The input/output unit 23 is configured to receive a user operation andsupply to the area control unit 22 the operation coordinate informationrelated to the screen in which the user operation is performed. Further,the input/output unit 23 is configured to receive an output image fromthe area control unit 22 and display a screen including the outputimage.

FIG. 6 shows an example of a viewing area pattern table 242A stored inthe viewing area pattern storage unit 242. The contents of the viewingarea pattern table 242A may be stored in the viewing area patternstorage unit 242 upon installation of an application program forimplementing the image acquisition unit 21, the area control unit 22,and the input/output unit 23. Subsequently, when a data change or a dataaddition to the viewing area pattern table 242A is provided, the datachange or the data addition is acquired from a server device or the likeso that the contents of the viewing area pattern table 242A may beupdated.

As shown in FIG. 6, the viewing area pattern table 242A includes dataitems of “VIEWING AREA PATTERN”, “NUMBER OF AREAS”, “RANGE OF IMAGE(INITIAL VALUE)”, “ARRANGEMENT PATTERN”, “STANDARD”, “ASSOCIATIONDISPLAY DIRECTION”, etc.

The “VIEWING AREA PATTERN” indicates identification information of eachof viewing area patterns. The “NUMBER OF AREAS” indicates the number ofviewing areas included in a corresponding one of the viewing areapatterns. The “RANGE OF IMAGE (INITIAL VALUE)” indicates an initialvalue of a range of an image displayed in each of the viewing areas inthe corresponding viewing area pattern. In a case of an omnidirectionalimage, the image is determined by a horizontal angle θ, a vertical angleφ, and a viewing angle α. Note that the range of the image is variedwhen moving or resizing of the image is performed by the user operation,but the varied range of the image is reflected in the current data.

The “ARRANGEMENT PATTERN” indicates how two or more viewing areas arearranged in a screen. In this embodiment, two or more candidates for thearrangement pattern are provided to enable a user to select a desiredone of the candidates. The selected arrangement pattern is reflected inthe current data.

Here, several examples of the arrangement pattern are explained. FIG. 7shows an example of the arrangement pattern. In this example, the screenis equally divided into two rows (up and down) in portrait orientationso that two viewing areas are provided. FIG. 8 shows an example of thearrangement pattern. In this example, the screen is divided into tworows (up and down) in portrait orientation so that two viewing areas areprovided (a lower viewing area is enlarged and an upper viewing area isreduced). FIG. 9 shows an example of the arrangement pattern. In thisexample, the screen is divided into two areas in portrait orientation sothat two viewing areas are provided (a small viewing area is arranged atthe upper right corner of the whole screen). FIG. 10 shows an example ofthe arrangement pattern. In this example, the screen is equally dividedinto three rows in portrait orientation so that three viewing areas arearrayed in the up/down direction of the screen. FIG. 11 shows an exampleof the arrangement pattern. In this example, the screen is equallydivided into two rows and two columns in portrait orientation so thatfour viewing areas are provided in a grid formation. FIG. 12 shows anexample of the arrangement pattern. In this example, the screen isdivided into five areas in portrait orientation, a large viewing area isarranged in the center of the screen, and two small viewing areas (rightand left) are arranged in each of an upper portion and a lower portionof the screen so that five viewing areas are provided. FIG. 13 shows anexample of the arrangement pattern. In this example, the screen isdivided into five areas in landscape orientation, a large viewing areais arranged in the center of the screen, and two small viewing areas (upand down) are arranged in each of a right portion and a left portion ofthe screen so that five viewing areas are provided.

Referring back to FIG. 6, the arrangement pattern may include a patternin which all of the viewing areas or some of the viewing areas areresized. In this case, the size of the viewing areas may be varied by auser operation. An operation to resize the viewing areas depends on anoperation interface (e.g., dragging an end portion of a viewing area)provided by an operating system (OS) of the information processingapparatus 2. The resized viewing areas are reflected in the currentdata. Further, the positions of all or some of the viewing areas may bemoved by a user operation.

In addition, the viewing area may be copied or deleted. The copiedviewing area may inherit the characteristics of the original viewingarea and may be reflected in the current data. The copied viewing areamay be newly registered in the viewing area pattern table 242A. Thecopied viewing area may be arranged in a vacant area in the screen oroverlaid on the existing viewing area. The arrangement may be set upsuch that the copied viewing area may not be overlaid on the existingviewing area. In this case, the position where the copied viewing areais arranged is adjusted relative to the existing viewing area.

The “STANDARD” indicates a standard viewing area when the associationrelationship between viewing areas is defined. The “ASSOCIATION DISPLAYDIRECTION” indicates a direction of association display of theassociation viewing area relative to the standard viewing area. Inmoving and resizing operations, each of the same direction and theopposite direction may be selected as the association display direction.The “ASSOCIATION DISPLAY DIRECTION” may be set up such that theselection is enabled for each of the association viewing areas.

Note that the viewing area pattern table 242A may be implemented byinserting corresponding codes in an image display program according tothe invention, instead of using a table form.

FIG. 14 shows an example of an area transition pattern table 243A storedin the area transition pattern storage unit 243. The contents of thetransition pattern table 243A may be stored in the area transitionpattern storage unit 243 upon installation of an application program forimplementing the image acquisition unit 21, the area control unit 22,and the input/output unit 23. Subsequently, when a data change or a dataaddition to the area transition pattern table 243A is provided, the datachange or the data addition is acquired from a server device or the likeso that the contents of the transition pattern table 243A may beupdated.

As shown in FIG. 14, the area transition pattern table includes dataitems of “AREA TRANSITION PATTERN”, “NUMBER OF AREAS”, “TRANSITIONDIRECTION”, “TRANSITION OF OPERATIONS”, “TARGET OF ASSOCIATION”, etc.

The “AREA TRANSITION PATTERN” indicates identification information ofeach of area transition patterns. The “NUMBER OF AREAS” indicates thenumber of viewing areas to which a corresponding one of the areatransition patterns is applied. The “TRANSITION DIRECTION” indicates adirectional relationship between viewing areas covered by a useroperation. Specifically, in the “TRANSITION DIRECTION”, a “LEFT<=>RIGHT”indicates that a user operation covers viewing areas which adjoin in aright/left direction, an “UP<=>DOWN” indicates that a user operationcovers viewing areas which adjoin in an up/down direction, and a“DIAGONAL” indicates that a user operation covers viewing areas whichadjoin in a diagonal direction (both in a right/left direction and in anup/down direction).

The “TRANSITION OF OPERATIONS” indicates information for treating as auser operation in a viewing area which a starting point of the touchingoperation performed belongs to when the user operation indicated by the“TRANSITION DIRECTION” is performed and the relationship between thestandard area and the association area is changed. The “TARGET OFASSOCIATION” indicates the contents of association in which movingquantities (moving direction, moving amount) and resizing quantities(resizing amount) are set.

Note that the area transition pattern table 243A may be implemented byinserting corresponding codes in the image display program according tothe invention, instead of using a table form.

FIG. 15 shows an example of the current data stored in the current datastorage unit 244. As shown in FIG. 15, the current data stored in thecurrent data storage unit 244 includes an image ID of an image selectedby a user, data items of the viewing area pattern selected from theviewing area pattern table 242A (FIG. 6) by the user, and data items ofthe area transition pattern corresponding to the viewing area pattern inthe area transition pattern table 243A (FIG. 14).

Next, FIG. 16 shows an imaging action performed using the imaging device1. As shown in FIG. 16, a user holds the imaging device 1 by hand andtakes an image by pressing the switch 1H3 (FIG. 2). The imaging device 1is configured to take an omnidirectional image covering all thedirections of the imaging device 1 by using the front imaging element1H1 (FIG. 2) and the rear imaging element 1H2 (FIG. 2).

FIG. 17 shows an example of images taken by the imaging device 1. Asshown in FIG. 17, a first image 17A is taken by the front imagingelement 1H1 of the imaging device 1 and a second image 17B is taken bythe rear imaging element 1H2 of the imaging device 1. The top and bottomof the second image 17B are opposite to those of the first image 17A.This is because a simplified optical system is used in this example. Inthe simplified optical system, light beams entering from lenses disposedin front of and behind the casing are reflected upward and downward bythe front and rear surfaces of a single mirror, and the front imagingelement 1H1 and the rear imaging element 1H2 are disposed on the opticalaxes of the reflected light beams. As shown in FIG. 17, the first andsecond images 17A and 17B are combined to generate an omnidirectionalimage 17C according to the equidistant cylindrical projection or theMercator projection.

FIG. 18 shows an example of generation of an omnidirectional image bythe omnidirectional image generation unit 13 (FIG. 5) of the imagingdevice 1. In the example of FIG. 18, a correspondence relationshipbetween the horizontal angle θ and the vertical angle φ with respect torespective positions of pixels of an image obtained from each of thefront imaging element 1H1 and the rear imaging element 1H2 is shown inthe top row portion of FIG. 18. The top and bottom of the left image inthe top row portion of FIG. 18 are opposite to the top and bottom of theright image, and the positive/negative of the angle φ of the left imageare reverse to that of the right image.

The pixels of the images in the top row portion of FIG. 18 aretransformed based on a look-up table to transform the angle θ and theangle φ into the Cartesian coordinates to generate images in which theangle θ ranges from 0 to 180 degrees as shown in the middle row portionof FIG. 18. Next, the two images in the middle row portion are combinedto generate an image in which the angle θ ranges from 0 to 360 degreesas shown in the bottom row portion of FIG. 18. Then, the image data ofthis image is decomposed into triangular polygon data, and the polygondata is associated with the angle θ and the angle φ so thatthree-dimensional (3D) data is created. The 3D data provides anomnidirectional image.

FIGS. 19A to 19C are diagrams showing a cutout of a display image. FIG.19A shows a relationship between an X-axis, a Y-axis, and a Z-axis. Athree-dimensional sphere CS is provided to define a spherical surface onwhich an omnidirectional image is virtually affixed. FIG. 19B shows thata virtual camera CAM is disposed within the three-dimensional sphere CSand cuts out of the spherical surface a display image (which correspondsto a partial image of the omnidirectional image) in a predetermined areaT on the spherical surface. FIG. 19C shows a field of view of thevirtual camera CAM. As shown in FIG. 19C, an angle which faces acrossthe ends of a diagonal line of the predetermined area T is defined asbeing a viewing angle α of the virtual camera CAM.

FIG. 20 shows a relationship between a position of the virtual cameraCAM and the origin of the three-dimensional sphere CS. As shown in FIG.20, the virtual camera CAM generally is moved back along the Z-axis (oralong the optical axis) by a distance d from the origin of thethree-dimensional sphere CS, and the viewing angle α is measured aroundthe position of the virtual camera CAM at the distance d. Here, it isassumed that, when the position of the virtual camera CAM at thedistance d lies inside the three-dimensional sphere CS, the positionswhere the two rays of the viewing angle α intersect with thethree-dimensional sphere CS are in agreement with the positions wherethe two rays of a viewing angle ω around the origin intersect with thethree-dimensional sphere CS, and when the position of the virtual cameraCAM at the distance d lies outside the three-dimensional sphere CS, thetwo rays of the viewing angle α are tangential to the three-dimensionalsphere CS. When the distance d is equal to 0, the viewing angle α is inagreement with the viewing angle co around the origin. The viewing angleω, the viewing angle α, and the distance d are controlled according to aresizing factor, and if the viewing angle ω, the viewing angle α, andthe distance d are controlled to reduce the image displayed in size, theoutline of the image displayed becomes circular.

FIG. 21 is a flowchart for explaining a process performed by theinformation processing apparatus 2.

Upon a start of the process shown in FIG. 21, the image selection unit221 of the information processing apparatus 2 provides a user with alist of the images stored in the image data storage unit 241, andreceives from the user a selection of a target image from among theimages of the list (step S101). The image data storage unit 241 storesan image ID of the selected target image into the current data storageunit 244 as a current data item.

Subsequently, the area setting unit 222 provides the user with a list ofviewing area pattern candidates from the viewing area pattern table 242Aof the viewing area pattern storage unit 242, and receives from the usera selection of a viewing area pattern from among the candidates of thelist (step S102). The area setting unit 222 determines whether theselection of the viewing area pattern is received from the user (stepS103). If it is determined that the selection of the viewing areapattern is received from the user (YES of step S103), the imageselection unit 221 stores the data items of the viewing area pattern andthe corresponding area transition pattern into the current data storageunit 244 as current data items.

In the present example, the viewing area pattern is selected by theuser. Alternatively, the information processing apparatus 2 may beconfigured so that the viewing area pattern is determined appropriatelybased on the features or characteristics of the image to be displayed.For example, the viewing area pattern may be determined based on theresult of analysis of a color tone or a time change of the image, theassigned feature information or identification information such as atag. In this case, it is conceivable that for an image lackingvariation, a viewing area pattern with a small number of viewing areasmust be determined in view of high visibility, and for an image havingvarious changes in different directions, a viewing area pattern with alarge number of viewing areas must be determined in view of thenecessity of checking the details.

Subsequently, the image assignment unit 223 performs assignment of theviewing areas based on the range (initial value) of the image of thedetermined viewing area pattern (step S104) and displays the targetimage (step S105). Note that the image assignment unit 223 determineswhether the information processing apparatus 2 is in portraitorientation or in landscape orientation based on an output signal of thestate sensor 2H4 (FIG. 4) and displays the target image such that thetop and the bottom of the image being displayed conform to thedetermined orientation of the information processing apparatus 2.

Subsequently, the operation determination unit 224 checks a useroperation on the screen (step S106) and determines whether the useroperation on the screen is received (step S107). If it is determinedthat the user operation is received (YES of step S107), the operationdetermination unit 224 determines a first viewing area which covers astarting point of a touching action of the user operation on the screenbased on the coordinate information of the user operation (step S108),and determines second viewing areas other than the first viewing area(step S109).

Subsequently, the operation determination unit 224 discriminates theuser operation in the first viewing area based on the number of touchportions and changes of the coordinate information of the user operation(step S110). Specifically, it is discriminated whether the useroperation is a slide (swipe) action, a pinch-out action, or a pinch-inaction. In this case, the operation determination unit 224 acquires anamount of the user operation in the first viewing area collectively.

Further, the operation determination unit 224 determines whether theoperation in the first viewing area is a valid operation (step S111). Ifit is determined that the operation in the first viewing area is a validoperation (YES of step S111), the operation determination unit 224determines changes (moving, resizing) to the image in the first viewingarea based on the discriminated operation and the acquired amount of theoperation (step S112). The operation determination unit 224 determineschanges (moving, resizing) to the images in the second viewing areasbased on the data item of the association display direction in theviewing area pattern of the stored current data (step S113).

At this time, when all of the viewing areas have the same size, theamount of changes (moving, resizing) to the image in the first viewingarea and the amount of changes to the images in the second viewing areasare the same. When the first and second viewing areas have differentsizes, the amounts of changes are varied depending on a ratio of atypical length of the first view area and a typical length of the secondviewing areas (e.g., a length of a diagonal line in a case of arectangular area, and a diameter of a circle in a case of a circulararea). For example, when the ratio of the typical length of the firstviewing area and the typical length of the second viewing areas is1:0.5, the amount of changes to the second viewing areas is half aslarge as the amount of changes to the first viewing area.

Further, in determining the changes to the first viewing area and thechanges to the second viewing areas at steps S112 and S113, if the useroperation covers both the first and second viewing areas, the operationdetermination unit 224 is configured to determine that a user operationwith respect to the earlier touched area of the first and second viewingareas is performed continuously. Namely, when the data item of thetransition direction in the area transition pattern of the current datais matched, the operation determination unit 224 determines the standardor association of the first and second viewing areas based on thetransition of operations of the current data and determines either thechanges to the first viewing area or the changes to the second viewingareas.

Subsequently, the image operation unit 225 applies the changes to eachof the image in the first viewing area and the images in the secondviewing areas based on the determined changes to the image in the firstviewing area and the determined changes to the second viewing areas, anddisplays the image in the first viewing area and the images in thesecond viewing areas concurrently (step S114).

Next, several examples of operations and display changes will bedescribed.

FIG. 22A and FIG. 22B show a case where the screen is equally dividedinto two rows (an upper viewing area and a lower viewing area) inportrait orientation and the association display direction of moving inthe upper viewing area is set to the direction opposite to the directionof moving in the lower viewing area. In this case, when a rightwardsliding operation is performed in the lower viewing area as shown inFIG. 22A, an image in the lower viewing area moves in a right directionwhile an image in the upper viewing area moves in a left direction asshown in FIG. 22B.

FIG. 23A and FIG. 23B also show the case where the screen is equallydivided into the two rows (the upper and lower viewing areas) inportrait orientation and the association display direction of moving inthe upper viewing area is set to the direction opposite to the directionof moving in the lower viewing area. In this case, when an upwardsliding operation is performed in the lower viewing area as shown inFIG. 23A, an image in the lower viewing area moves in an upwarddirection while an image in the upper viewing area moves in a downwarddirection as shown in FIG. 23B.

FIG. 24A and FIG. 24B also show the case where the screen is equallydivided into the two rows (the upper and lower viewing areas) inportrait orientation and the association display direction of moving inthe upper viewing area is set to the direction opposite to the directionof moving in the lower viewing area. In this case, when a slidingoperation in a right upward diagonal direction is performed in the lowerviewing area as shown in FIG. 23A, an image in the lower viewing areamoves in the right upward diagonal direction while an image in the upperviewing area moves in a left downward diagonal direction as shown inFIG. 24B.

FIG. 25A and FIG. 25B show a case where the screen is equally dividedinto two columns (a right viewing area and a left viewing area) inlandscape orientation and the association display direction of moving inthe right viewing area is set to the direction opposite to the directionof moving in the left viewing area. In this case, when a rightwardsliding operation is performed in the left viewing area as shown in FIG.25A, an image in the left viewing area moves in a right direction whilean image in the right viewing area moves in a left direction as shown inFIG. 25B.

FIG. 26A and FIG. 26B also show the case where the screen is equallydivided into the two columns (the right and left viewing areas) inlandscape orientation and the association display direction of moving inthe right viewing area is set to the direction opposite to the directionof moving in the left viewing area. In this case, when an upward slidingoperation is performed in the left viewing area as shown in FIG. 26A, animage in the left viewing area moves in an upward direction while animage in the right viewing area moves in a downward direction as shownin FIG. 26B.

FIG. 27A and FIG. 27B also show the case where the screen is equallydivided into the two columns (the right and left viewing areas) inlandscape orientation and the association display direction of moving inthe right viewing area is set to the direction opposite to the directionof moving in the left viewing area. In this case, when a slidingoperation in a right upward diagonal direction is performed in the leftviewing area as shown in FIG. 27A, an image in the left viewing areamoves in the right upward diagonal direction while an image in the rightviewing area moves in a left downward diagonal direction as shown inFIG. 27B.

FIG. 28A and FIG. 28B show the case where the screen is equally dividedinto the two rows (the upper and lower viewing areas) in portraitorientation and the association display direction of moving in the upperviewing area is set to the direction opposite to the direction of movingin the lower viewing area. In this case, when a sliding operation in aright upward diagonal direction is performed in a range from the lowerviewing area to the upper viewing area as shown in FIG. 28A, an image inthe lower viewing area moves in the right upward diagonal directionwhile an image in the upper viewing area moves in a left downwarddiagonal direction as shown in FIG. 28B.

FIG. 29A and FIG. 29B show a case where the screen is divided into twoareas (a small viewing area at an upper right corner portion of thescreen and a large viewing area spreading over the whole screen) inportrait orientation and the association display direction of moving inthe small viewing area is set to the direction which is the same as thedirection of moving in the large viewing area. In this case, when arightward sliding operation is performed in the large viewing area asshown in FIG. 29A, an image in the large viewing area moves in a rightdirection and an image in the small viewing area moves in the rightdirection as shown in FIG. 29B.

FIG. 30A and FIG. 30B show a case where the screen is equally dividedinto the two rows (the upper and lower viewing areas) in portraitorientation and the association display direction of resizing in theupper viewing area is set to the direction which is the same as thedirection of resizing in the lower viewing area. In this case, when apinch-out operation is performed in the lower viewing area as shown inFIG. 30A, an image in the lower viewing area is expanded and an image inthe upper viewing area is also expanded as shown in FIG. 30B.

FIG. 31A and FIG. 31B show a case where the screen is divided into thetwo areas (the small and large viewing areas) in portrait orientationand the association display direction of resizing in the upper viewingarea is set to the direction which is the same as the direction ofresizing in the lower viewing area. In this case, when a pinch-outoperation is performed in the lower viewing area as shown in FIG. 31A,an image in the lower viewing area is expanded and an image in the upperviewing area is also expanded as shown in FIG. 31B.

As described in the foregoing, the image display system according to theinvention is capable of providing a user interface which enablesflexible operations to be performed when displaying an image in aplurality of viewing areas.

The image display system according to the invention is not limited tothe above-described embodiments, and variations and modifications may bemade without departing from the scope of the present invention.

The image acquisition unit 21 in the foregoing embodiment is an exampleof “image acquisition unit.” The area setting unit 222 in the foregoingembodiment is an example of “area setting unit.” The image assignmentunit 223 in the foregoing embodiment is an example of “image assignmentunit.” The operation determination unit 224 in the foregoing embodimentis an example of “operation determination unit.” The image operationunit 225 in the foregoing embodiment is an example of “display controlunit.” The area control unit 22 in the foregoing embodiment is anexample of “displaying unit.” The area control unit 22 in the foregoingembodiment is an example of “image portion changing unit.”

What is claimed is:
 1. An image display system comprising one or morehardware processors configured to: acquire an omnidirectional image; seta plurality of viewing areas to which the image is to be assigned, theplurality of viewing areas including a first viewing area and a secondviewing area other than the first viewing area; display a first portionof the omnidirectional image in the first viewing area and display asecond portion of the omnidirectional image in the second viewing area;receive an operation in the first viewing area; and change, based on theoperation, the first portion displayed in the first viewing area and thesecond portion displayed in the second viewing area, wherein when a sizeof the first viewing area is equal to a size of the second viewing area,an amount of change applied to the first portion displayed in the firstviewing area is equal to an amount of change applied to the secondportion displayed in the second viewing area, and wherein when the sizeof the first viewing area differs from the size of the second viewingarea, the amount of change applied to the first portion displayed in thefirst viewing area and the amount of change applied to the secondportion displayed in the second viewing area are changed in accordancewith a ratio between the size of the first viewing area and the size ofthe second viewing area.
 2. The image display system according to claim1, wherein the one or more hardware processors are configured to set anassociation display direction between the plurality of viewing areas andthe one or more hardware processors are configured to change the secondportion in the second viewing area based on the association displaydirection in response to a change of the first portion in the firstviewing area.
 3. The image display system according to claim 2, whereinthe association display direction includes an association displaydirection of moving the first portion and the second portion and anassociation display direction of resizing the first portion and thesecond portion, and the association display direction includes a samedirection and an opposite direction.
 4. The image display systemaccording to claim 2, wherein the one or more hardware processors areconfigured to determine an amount of change based on a ratio of atypical length of the first viewing area and a typical length of thesecond viewing area.
 5. The image display system according to claim 1,wherein, when the operation covers the first and second viewing areas,the one or more hardware processors are configured to determine that anoperation with respect to an earlier touched area of the first andsecond viewing areas is performed continuously.
 6. The image displaysystem according to claim 1, wherein the operation for changing thefirst portion is a swipe operation, a pinch-out operation, or a pinch-inoperation.
 7. The image display system according to claim 6, wherein thechange of the first portion by the swipe operation is movement of thefirst portion.
 8. The image display system according to claim 6, whereinthe change of the first portion by the pinch-out operation or thepinch-in operation is resizing of the first portion.
 9. The imagedisplay system according to claim 1, wherein the first portion isdefined based on a first horizontal angle in a horizontal direction anda first vertical angle in a vertical direction of the omnidirectionalimage and the second portion is defined based on a second horizontalangle, which is different from the first horizontal angle, in thehorizontal direction and a second vertical angle, which is equal to thefirst vertical angle, in the vertical direction of the omnidirectionalimage.
 10. The image display system according to claim 1, wherein theone or more hardware processors are configured to determine the amountof change applied to the first portion and the amount of change appliedto the second portion based on a ratio of a length of the first viewingarea and a length of the second viewing area.
 11. An image displaymethod performed by an information processing apparatus, the methodcomprising: acquiring an omnidirectional image; setting a plurality ofviewing areas to which the image is to be assigned, the plurality ofviewing areas including a first viewing area and a second viewing areaother than the first viewing area; displaying a first portion of theomnidirectional image in the first viewing area and display a secondportion of the omnidirectional image in the second viewing area;receiving an operation in the first viewing area; and changing, based onthe operation, the first portion displayed in the first viewing area andthe second portion displayed in the second viewing area, wherein when asize of the first viewing area is equal to a size of the second viewingarea, an amount of change applied to the first portion displayed in thefirst viewing area is equal to an amount of change applied to the secondportion displayed in the second viewing area, and wherein when the sizeof the first viewing area differs from the size of the second viewingarea, the amount of change applied to the first portion displayed in thefirst viewing area and the amount of change applied to the secondportion displayed in the second viewing area are changed in accordancewith a ratio between the size of the first viewing area and the size ofthe second viewing area.
 12. The image display method according to claim11, wherein the operation for changing the first portion is a swipeoperation, a pinch-out operation, or a pinch-in operation.
 13. The imagedisplay method according to claim 12, wherein the changing of the firstportion by the swipe operation is movement of the first portion.
 14. Theimage display method according to claim 12, wherein the changing of thefirst portion by the pinch-out operation or the pinch-in operation isresizing of the first portion.
 15. A non-transitory recording mediumstoring a program that causes an image display system to execute aprocess, the process comprising: acquiring an omnidirectional image;setting a plurality of viewing areas to which the image is to beassigned, the plurality of viewing areas including a first viewing areaand a second viewing area other than the first viewing area; displayinga first portion of the omnidirectional image in the first viewing areaand display a second portion of the omnidirectional image in the secondviewing area; receiving an operation in the first viewing area; andchanging, based on the operation, the first portion displayed in thefirst viewing area and the second portion displayed in the secondviewing area, wherein when a size of the first viewing area is equal toa size of the second viewing area, an amount of change applied to thefirst portion displayed in the first viewing area is equal to an amountof change applied to the second portion displayed in the second viewingarea, and wherein when the size of the first viewing area differs fromthe size of the second viewing area, the amount of change applied to thefirst portion displayed in the first viewing area and the amount ofchange applied to the second portion displayed in the second viewingarea are changed in accordance with a ratio between the size of thefirst viewing area and the size of the second viewing area.
 16. Thenon-transitory recording medium according to claim 15, wherein theoperation for changing the first portion is a swipe operation, apinch-out operation, or a pinch-in operation.
 17. The non-transitoryrecording medium according to claim 16, wherein the changing of thefirst portion by the swipe operation is movement of the first portion.18. The non-transitory recording medium according to claim 16, whereinthe changing of the first portion by the pinch-out operation or thepinch-in operation is resizing of the first portion.